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Abstract 

For  the  analysis  and  optimisation  of  Decision  Processes  of  small  teams  (2-20  persons)  as  well 
as  individual  decision  makers  an  Analysis  Framework  is  under  development  within  Task 
Group  006  (Modelling  of  Organisations  and  Decision  Architectures)  of  the  NATO  RTO/IST 
Panel. 

A  team  from  6  nations,  The  Netherlands,  France,  The  United  States,  Norway,  the  Czech 
Republic  and  Canada  are  developing  this  framework  which  is  comprised  of  models  on  decision 
processes  and  the  functioning  of  teams,  a  analysis  area  model  for  analysis  and  a  procedural 
description  on  how  to  apply  the  modelling  framework  to  specific  problem  domains.  The  model 
on  decision  processes  is  based  upon  some  basic  axioms  on  human  decision  making.  The  model 
on  team  functioning  identifies  elementary  behavioural  functions  related  to  teams  and  an 
integration  model  is  proposed  between  team  functions  and  human  decision  making.  The  total 
analysis  and  optimisation  framework  also  includes  formal  description  techniques  with  which 
the  modelling  and  analysis  results  can  be  captured  and  described  and  a  set  of  measures  of  merit 
with  which  decision  making  and  team  functioning  can  be  measured. 

This  paper  describes  the  applicable  problems  domains  and  the  current  state  of  development  of 
the  Analysis  Framework. 

1.  Introduction 

The  concepts  presented  in  this  paper  reflect  the  current  state  of  work  of  the  international 
NATO  1ST  TG0061  Task  Group  on  Modelling  of  Organisation  and  Decision  Making.  TG006 
aims  to  propose  a  methodology  for  describing  and  analysing  organisations  with  respect  to 
decision  processes,  taking  into  account  human  factors.  This  document  takes  a  first  step  at 
developing  such  a  methodology.  It  introduces  a  set  of  models  and  tools,  enclosed  in  a 
framework  structure  that  can  be  applied  to  systematically  assess  the  structure  and  performance 
of  an  organisation.  It  takes  into  account  various  aspects  of  team  decision  making.  In  the  end  it 
aims  to  provide  a  way  to  develop  recommendations  on  how  to  improve  decision  making  in  an 
organisation  by  means  of  additional  support  tools  or  organisational  adjustments. 


1  The  author  wishes  to  thank  the  members  of  the  TG006  for  their  support  in  developing  this  framework  and  also 
G.  Klein,  M.  Endsley  and  R.  Rousseau  for  their  constructive  review  of  an  earlier  version  of  this  document. 
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1.1  Terms  and  Concepts 

Because  decision  making  can  only  be  described  in  fairly  abstract  terms,  we  need  to  avoid 
confusion  of  terminology.  Generally,  the  term  ‘decision  making’  is  used  as  an  umbrella  term 
(Klein,  1992)  for  all  activities  involved  in  choosing  one  solution  in  the  total  solution  space  of  a 
problem.  We  also  adopt  this  stance,  but  we  are  going  to  diverge  a  little  from  the  standard 
terminology  from  here  on.  We  would  like  to  introduce  a  distinction  between  decision  making, 
decision  process  and  option  selection  (or  decide  on  action).  If  we  talk  about  decision  making, 
we  adopt  the  general  definition: 

decision  making 

Decision  making  refers  to  the  act  of  judging,  evaluating,  choosing,  selecting,  picking,  and 
resolving  options.  Decision  making  concerns  known  alternatives,  goals  and  missions. 

‘Decision  making’  thus  refers  to  the  entire  act  in  a  single  term.  If  we  describe  decision  making 
as  an  act  that  consists  of  several  sub-processes,  then  we  refer  to  it  as  a  decision  process.  We 
define  a  decision  process  as  follows: 

decision  process 

The  decision  process  contains  a  sequence  of  activities  that  as  a  whole  constitute  decision 
making.  These  activities  include  such  activities  as  observing  the  environment,  situational 
awareness,  decide  on  action.  A  decision  process  model  therefore  refers  to  an  abstract 
representation  of  all  the  processes  that  are  part  of  decision  making. 

When  decision  making  is  viewed  as  a  process,  then  option  selection,  the  actual  selection  of  the 
decision,  is  a  part  of  the  entire  decision  process.  There  are  other  processes  that  are  contained 
within  the  decision  process,  but  which  are  not  part  of  the  actual  option  selection.  These  are 
nevertheless  necessary  prerequisites  for  option  selection.  For  instance,  information  or  data 
gathering  is  such  an  elementary  process.  Often,  the  term  option  selection  is  taken  be 
synonymous  with  decision  making.  We  state  that  option  selection  is  a  sub-process  of  the 
decision  process.  This  concurs  with  what  Klein  and  others  (Orasanu,  1993)  have  observed  in 
research. 

2.  Goals  and  Context 

2.1  TG006  Goals 

The  IST-TG006  project  aims  to  study  decision  processes  within  complex  and  demanding 
military  operations.  Research  on  decision  making  involves  identifying  the  specific 
characteristics  of  the  environments  where  decision  making  takes  place,  modelling  the 
organisational  structure  of  decision  process  and  particularly,  studying  the  human  factors  that 
are  involved  in  decision  making.  The  assumption  is  that  by  studying  decision  making 
organisations  and  modelling  the  decision  making  process  we  can  not  only  assist  organisations 
in  making  better  decisions,  but  also  contribute  to  develop  better  training  and  support  tools. 
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This  general  statement  of  work  may  quickly  lead  to  an  fairly  academic  exercise  if  we  do  not 
concentrate  on  establishing  beforehand  how  the  knowledge  gained  from  this  study  is  going  to 
be  applied.  Therefore  the  results  gained  from  this  research  are  applied  to  several  case  studies. 
These  examples  of  military  decision  making  will  enable  us  to  evaluate  and  refine  our 
modelling  efforts. 

2.2  Application  area  of  TG006 

Our  assumption  is  that  decisions  are  always  made  within  the  context  of  fulfilling  a  specific 
task.  Since  the  characteristics  of  a  task  (or  rather:  goal)  directs  the  decision  processes,  it  is 
essential  that  we  specify  on  which  kinds  of  tasks  this  research  is  aimed  at.  Two  criteria  have 
been  used  in  selecting  the  application  areas  of  this  TG006  work: 

•  Small  teams 

We  restrict  ourselves  to  small  teams,  i.e.  ranging  from  a  single  person  to  a  team  of  about 
twenty  members.  Larger  organisations  are  harder  to  study,  because  they  usually  involve  more 
than  one  particular  decision  making  processes  which  are  more  or  less  independent  and  take 
place  at  different  levels.  In  general  one  could  assume  that  the  total  decision  making  in  these 
larger  organisations  can  be  decomposed  into  smaller  ones  for  which  the  results  of  our  study 
should  be  applicable  again.  However  we  will  not  make  that  explicit  assumption  and  only 
restrict  ourselves  to  small  teams. 

•  Fairly  structured  environments 

A  peace-establishing  process  (e.g.  the  middle  East)  is  a  good  example  of  a  less  structured 
problem  domain.  It  involves  a  highly  complex  and  unclear  problem  domain  (what  are  the 
factors  that  contribute  to  the  war)  and  solution  space  (what  can  we  do  about  it?).  Due  to  these 
factors  these  types  of  scenario’s  to  are  hard  to  characterise.  We  have  not  aimed  our  results  to 
be  applicable  in  such  a  domain.  Initially  we  restrict  our  modelling  activities  to  fairly  structured 
problem  domains  in  order  to  get  a  grip  on  contributing  factors  and  issues  that  affect  decision 
making  performance.  Later  on  we  may  consider  the  applicability  in  other  problem  domains.  C2 
tasks,  like  threat  evaluation  of  radar  contacts,  is  a  complex  procedure  as  well,  but  has  a 
relatively  small  solution  space  (if  the  contact  is  a  threat  we  may  need  to  engage  it,  if  it  is  not 
we  do  not  need  to  engage  it).  This  makes  C2  tasks  more  suitable  for  our  purposes. 

Three  decision  making  scenarios  (case-studies)  have  been  selected  which  meet  these 
requirements: 

1.  Ship-based  C2, 

2.  Logistics 

3.  Weather  forecasting. 

These  case-studies  represent  specific  organisational  structures,  domains  and  goals.  Each  of  the 
cases  will  have  its  own  characteristics  in  terms  of  qualitative  identifiers  (measures  that  signify 
how  well  a  task  has  been  performed).  Within  the  TG006  work  these  three  case  studies  will  be 
used  to  develop,  test  and  refine  the  Analysis  Framework.  It  should  be  mentioned  that  one 


CCRP  Symposium  2002,  June  2002,  Monterey,  CA 


Page  4 


Hans  E.  Keus  A  Framework  for  Analysis  of  Decision  Processes  in  Teams 


important  pragmatic  reason  of  choosing  these  cases  was  because  they  were  available  through 
the  personal  expertise  of  the  TG006  members. 

2.3  Intentions  of  this  paper 

This  paper  aims  to  provide  insight  in  the  current  state  of  the  Analysis  Framework  as  it  is 
developed  within  TG006.  It  describes  the  most  important  concepts  and  models  and  it  ends  with 
a  summary  of  our  current  research  activities  to  enhance  and  complete  the  framework.  It  is  by 
no  means  a  finished  product  and  it  will  be  fine  tuned  during  the  last  year  of  the  TG006  study 
year  (ending  mid  2003). 


3.  Decision  Modelling  Framework 

3.1  General  Approach 

In  figure  3.1  the  general  approach  we  have  used  is  shown.  We  will  use  this  drawing  to  explain 
our  approach  to  the  analysis  of  team  decision  making  processes. 


Figure  3.1  The  General  Approach. 


First  of  all  we  have  the  decision  making  team  (DMT)  which  operates  in  a  certain  operational 
context  which  is  part  of  the  real  world.  We  will  assume  this  DMT  has  a  specific  goal  (or 
mission)  to  fulfil.  The  overall  performance  of  the  DMT  is  expressed  by  a  measure  of 
effectiveness  (MoE).  Inside  the  team  we  have  measures  of  performance  (MoP)  with  respect  to 
different  aspects  of  team  functioning  and  decision  making.  The  MoPs  result  in  the  observable 
and  measurable  MoE  on  the  ‘outside’  of  the  team.  The  need  for  a  decision  performance 
analysis  originates  from  discrepancies  between  the  desired  MoE  or  the  internal  MoPs  and  the 
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actual  observed  ones  an.  Of  course  there  can  be  other  reasons  for  analysing  a  teams  decision 
making  process,  like  for  instance  a  desire  to  reduce  human  involvement  in  the  decision  making 
process  (manning  reduction  is  a  well-known  reason  in  the  maritime  context).  For  our  purposes 
it  is  important  we  have  a  well  defined  purpose  of  the  analysis  of  the  DMT  since  this  helps  to 
direct  the  analysis  process. 

The  analysis  takes  place  using  the  Analysis  Framework  (AF)  we  developed.  Conceptually  the 
AF  consist  of  a  number  of  models.  These  are  the  models  as  shown  in  Figure  3.1:  The  Decision 
Process  model,  the  Team  Functioning  Model,  the  Analysis  Area  Model,  the  Factor  Model  and 
the  Generic  Solution  Model.  These  models  will  be  explained  in  this  paper. 

One  of  the  elements  of  the  AF  is  a  stepwise  approach  which  directs  the  analysis  process  how  to 
use  the  concepts  described  in  the  models.  From  this  stepwise  approach  analyses  and 
measurements  are  conducted  of  the  DMT  and  its  environment,  here  denoted  by  the  operational 
context.  This  is  shown  in  Figure  3.1  as  the  analysis  and  measurements  arrow.  This  step 
provides  the  data  and  information  required  to  study  MoPs  of  the  DMT,  to  analyse  them  and  to 
formulate  solutions  and  recommendations  for  the  team  and  if  applicable  for  the  operational 
context  too.  Linked  to  the  essential  models  are  some  more  specialised  models  and  formal 
description  techniques  we  will  refer  to  in  the  paper.  Figure  3.1  illustrates  the  general  approach 
and  places  the  concepts  of  the  analysis  framework  in  a  context. 

3.2  Four  Conceptual  Layers 

We  distinguish  four  levels  of  abstraction  in  the  analysis  framework  (see  Figure  3.1). 

Generic 


Concepts 


Effects 


Causes 


Solutions 


Figure  3.1:  The  conceptual  levels  in  the  analysis  framework. 

The  first  level  contains  the  generic  concepts  we  use  in  our  framework.  Basically  these  are 
descriptions  of  our  area  of  attention:  decision  processes  of  organisations  and  teams.  This  is 
why  use  two  essential  models  here:  the  decision  process  model  and  the  team  functioning 
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model.  They  provide  us  with  an  understanding  of  what  decision  making  is  and  what  team 
specific  functions  are. 

The  decision  process  model  describes  the  basic  steps  that  are  made  in  decision  making.  There 
are  two  basic  categories  of  decision  making  models  (Lipshitz,  1993):  process  models  and 
typological  models.  The  former  describes  the  order  of  processes  in  which  decisions  are  made 
(like  Klein’s  model  (Klein,  1993),  whereas  the  latter  classifies  decision  processes  into  types 
and  describes  the  situation  these  processes  are  used  in  (like  the  Rasmussen  model  of  skill-,  rule 
and  knowledge  based  behaviour  (Rasmussen,  1983).  For  our  purposes  we  use  a  process  model, 
because  we  need  to  be  able  to  pinpoint  the  specific  events  in  the  decision  process  at  which 
improvement  can  be  achieved. 

The  team  functioning  model  contains  specific  team  functions  that  occur  in  team  decision 
making.  Together  the  decision  process  model  and  the  team  functioning  model  make  up  the 
team  decision  process  model.  These  models  are  described  in  more  detail  in  the  next  section. 

The  effect  level  deals  with  the  observable  characteristics  of  how  well  the  DMT  functions.  Here 
we  have  the  measures  of  performance  of  specific  team  decision  makings  aspects  and  of  the 
overall  measure  of  effectiveness  of  the  team.  We  call  these  the  Measures  of  Merit  (MoM). 
They  are  characteristics  which  can  be  measured  and  analysed.  Therefore  this  level  also 
contains  specific  analysis  methods  related  to  various  aspects  of  decision  making.  Central  to  this 
level  is  the  analysis  area  model  which  contains  the  areas  of  interest  to  study  decision 
processes.  The  MoMs  for  decision  making  and  team  functioning  are  all  related  to  one  or  more 
of  these  areas  of  attention.  Since  these  models  are  generic,  they  only  contain  measures  that  are 
ubiquitous,  and  independent  for  specific  application  domains.  These  generic  measures  are 
intended  to  be  applicable  to  all  scenarios  as  described  in  section  2.2. 

The  causes  level  contains  factors  (causes)  which  influence  the  decision  making  process,  and  is 
divided  into  organisational,  human,  informational  and  operational  factors.  Each  of  these  factors 
can  contribute  to  degradation  (or,  for  that  matter,  improvement)  of  the  decision  making 
process. 

The  solutions  level,  finally,  contains  generic  designs  or  measures  for  organisational  and 
informational  improvements  and  a  generic  architecture  for  decision  support  tools. 

The  following  sections  describe  the  various  sub-models  included  in  the  framework,  starting 
with  the  generic  level. 

4.  The  Conceptual  Level 

4.1  The  Decision  Process  Model 

4.1.1  Premises 

One  of  the  essential  models  in  the  framework  is  the  decision  process  model.  At  the  basis  of  the 
DPM  there  are  four  concepts  which  can  be  traced  back  to  the  OODA  loop  (Observe-Orient- 
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Decide- Act)  introduced  by  Boyd.  The  OODA  theory  was  specifically  developed  for  pilots  in 
combat  aircraft  where  fast  decision  making  is  essential  for  success.  We  will  not  copy  the  plain 
sequence  of  the  four  tasks  that  Boyd  has  introduced  but  we  use  these  tasks  and  add  to  them 
other  essential  elements  in  the  process  of  decision  making.  The  DPM  is  centered  around  the 
following  basic  premises: 

1)  Our  first  premise  is  that  in  order  to  decide  on  any  possible  action  the  decision  maker  needs 
three  basic  ingredients: 

a)  skills; 

b)  information; 

c)  and  understanding. 

He  should  have  an  understanding  of  the  situation  he  is  in.  In  order  to  achieve  that 
understanding  information  is  required.  This  can  be  external  data  (through  the  process  of 
data  gathering)  or  it  can  be  internal  information  already  available  to  the  decision  makers  or 
the  team  (such  as  the  knowledge  and  experience  of  the  team  members).  The  skills  applied 
in  order  to  decide  on  a  possible  course  of  action  result  an  action.  In  principle  this  is  just  an 
acknowledgement  of  the  four  essential  tasks  identified  by  Boyd  enriched  with  the 
capabilities  essential  in  decision  making:  skills,  experience. 


2)  The  second  premise  is  very  much  related  to  the  first  one  and  is  related  to  perception.  It  is 
concerned  with  the  fact  that  information  and  understanding  that  information  is  crucial  in 
decision  making.  In  order  to  reason  about  events,  data,  information  and  to  understand  it  we 
assume  that  there  are  three  essential  different  ‘world’  representations  relevant  to  the 
process  of  decision  making. 

a)  The  first  is  the  real  world  itself,  or  more  precise  the  actual  operational  context  which  is 
relevant  to  the  team  and  in  which  events  take  place. 

b)  The  second  is  the  presentation  of  (or  parts  of)  the  operational  context  for  the  benefit  of 
the  decision  makers:  the  presented  world  or  model.  Usually  this  is  graphical,  audio, 
video  or  other  data  and  information. 

c)  The  third  is  the  interned  model  or  mental  model  in  the  minds  of  the  decision  makers. 
This  is  the  interpretation  of  the  presented  model  using  internal  knowledge. 

Especially  when  dealing  with  team  decision  making  differences  between  mental  models  of 
individual  decision  makers  can  be  the  cause  of  wrong  decision  making.  Although  it  is  not 
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required  that  the  mental  models  must  be  equal  but  they  should  at  least  not  be  conflicting. 
In  Figure  4. 1  the  three  ‘world’  models  are  shown. 

3)  The  third  premise  is  that  decision  making  is  not  a  one  directional  process.  It  is  comprised 
of  continuous  switching  between  states  of  information  gathering,  information 
interpretation  and  assessment  of  this  information  in  order  to  make  the  actual  decision.  In 
the  process  of  moving  from  one  state  to  the  other  inferencing,  deduction,  pattern 
recognition  and  a  lot  of  other  cognitive  processes  may  play  an  important  role.  Our  purpose 
is  not  to  specify  and  explain  the  cognitive  processes  involved  but  to  allow  referencing  to 
them  from  our  model.  There  are  already  specific  models  made  by  others  which  already 
address  these  processes. 


4.1.2  The  DPM 

When  we  combined  the  elements  contained  in  each  of  the  three  axioms  we  can  construct  the 
Decision  Process  Model  as  shown  in  Figure  4.1. 


Primary  flow  Mental  model  Linking  assessment  Closing  the 

and  decision  Control  loop 


Figure  4. 1  The  Decision  Process  Model 
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Figure  4.1  is  a  high  level  representation  of  the  decision  making  process.  There  are  two  reasons 
for  the  high  level  of  abstraction  of  the  model. 

•  Using  a  general  model  allows  for  problem- specific  detailing 

Since  decision  making  strategies  can  differ  significantly  for  different  problem  domains  it  is 
impossible  to  capture  all  types  of  procedural  decision  making  itself  in  one  model.  Therefore  we 
aim  at  a  level  of  detail  for  general  framework  which  is  general  enough  for  many  problem 
domains. 

•  The  model  is  intended  to  identify  the  major  processes  involved  in  decision  making 

This  model  is  an  integral  part  of  our  proposed  framework.  Its  main  purpose  is  to  identify  the 
major  elements  of  the  decision  process.  This  enables  the  analysis  process  to  trace  all  relevant 
factors  which  influence  the  quality  of  the  decision  making  process.  It  is  not  meant  to  explain 
the  actual  decision  making  itself.  If  necessary  specialised  models  can  be  used  to  study  those 
aspects  (like  the  specific  available  models  of  Klein,  Rasmussen,  Endsley,  Noble,  Wearn,  etc.). 

4.1.3  Explanation  of  the  DPM 

The  process  which  provides  the  presented  world  or  model  we  have  called  data  gathering.  This 
usually  is  a  process  which  collects,  correlated,  fuses  and  presents  data.  The  data  can  come  from 
observation  devices  but  also  from  information  systems  or  human  observation. 

Observations  of  the  environment  lead  to  a  presented  world ,  i.e.  a  reflection  of  the  way  the 
operational  context  is  seen  by  all  information  sources.  This  presented  world  together  with 
internal  knowledge,  skill,  experience  provides  the  basis  for  creating  a  situational  awareness. 
This  process  we  have  called  understanding  the  situation. 

It  consist  of  several  sub-processes  and  states.  One  of  the  main  difficulties  operations  encounter 
in  creating  situation  awareness  is  a  lack  of  information  and  lack  of  confidence.  By  actively 
gathering  additional  information  one  can  strive  to  resolve  this  shortage.  The  request  connection 
serves  as  a  placeholder  for  all  activities  the  decision  maker  initiates  to  gather  information  that 
supports  his  situation  awareness  and  increases  his  level  of  confidence.  The  control  connection 
stops,  starts  and  controls  the  information  gathering  process.  This  connection  is  necessary  for 
e.g.  filtering  of  information.  The  request  connection  from  the  gathering  procedure  to  the  world 
represents  the  actual  collecting  of  the  information. 

Having  a  situational  awareness  essentially  means  that  the  decision  maker  has  interpreted  the 
world  in  terms  of  tactical  scenario’s,  assumptions  about  what  is  happening  and  projections  of 
causes  of  action  that  are  likely  to  happen.  Since  it  is  the  operator’s  understanding  of  the 
situation  this  interpretation  may  be  subject  to  errors.  Therefore  the  operator  may  try  to  validate 
his  situational  awareness  by  gathering  more  information  (i.e.  by  using  the  feedback  loop  to  the 
information  gathering  process),  or  might  just  use  his  present  understanding  for  the  actual 
decide-on-action  part. 

The  decide-on-action  process  consists  of  several  key-elements,  which  deal  with  selecting  an 
option.  At  this  stage  the  operator  has  an  interpretation  of  the  situation.  Aside  from  this 
situational  awareness,  the  operator  has  fixed  mission  goals  (protect  high  value  unit,  defend  own 
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ship,  protect  crew,  etc..).  Which  strategy  is  used  to  chose  a  particular  decision  (e.g.  select  an 
option  or  action)  depends  on  the  situation,  the  preference  of  the  operator  and  the  type  of  the 
problem  and  rules  of  engagements  in  military  situations.  In  general  the  decision  process  is 
guided  by  mission  goals,  plans  and  risk  assessments.  When  evaluating  a  goal  related  action  the 
operator  will  aim  to  validate  his  plan  for  action  within  his  existing  situation  awareness  or  strive 
to  match  the  elements  of  his  plan  with  observations  from  the  actual  world  (by  calling  upon  the 
information  gathering  process).  For  this  reason  a  direct  coupling  exists  in  the  model  between 
the  situation  awareness  process  and  the  decide-on-action  process.  Finally,  the  goal  related 
action  or  script  execution  resulting  from  the  selected  decision  has  a  certain  impact  on  the  ‘real 
world’,  which  in  turn  can  be  observed  by  means  of  the  information  gathering  process. 

It  is  well  established  that  the  process  of  making  decisions  is  not  a  single  sequential  process,  but 
rather  a  continuing  cycle  of  processes,  including  evaluation,  verification,  matching  and 
feedback  procedures.  Even  though  the  model  above  suggests  that  the  processes  involved  in 
decision  making  are  executed  sequentially,  these  processes  are  more  likely  to  be  executed 
concurrently  and  recursively.  One  of  the  features  of  our  model  to  capture  this  kind  of  decision 
making  strategy  is  the  inclusion  of  the  information  requests  and  the  control  loops. 

4.1.4  Relation  to  other  Decision  Process  Models 

The  decision  process  model  (Figure  4.1)  is  a  generic  representation.  Essentially,  it  can  be 
replaced  by  any  other  model  that  represents  decision  making,  be  it  either  more  detailed, 
abstract  or  even  having  a  fundamentally  different  take  on  decision  making,  as  long  as  it  is  in 
agreement  with  the  other  models  in  the  framework.  The  presented  decision  process  model  is 
designed  to  cover  the  majority  of  decision  process  models.  If  the  use  of  a  more  detailed  or 
specific  model  is  needed  this  can  be  derived  from  or  projected  to  our  model.  Within  TG006  an 
analysis  has  been  performed  in  which  we  derived  more  specific  models  from  our  DPM. 

4.2  Team  Functioning 

The  decision  process  model  as  described  in  section  4.1  is  a  model  of  decision  making 
irrespective  of  the  number  of  (human)  decision  makers.  It  focuses  only  on  identifying  and 
relating  the  major  elements  of  a  decision  making  process.  Since  decision  making  within  teams 
differs  from  individual  decision  making  another  model  is  present  in  our  framework,  the  team 
functioning  model,  which  identifies  the  specific  characteristics  of  decision  making  which  are 
involved  when  taking  a  team  or  an  organisation  into  account.  We  define  a  team  as  follows: 

A  team  is  an  organised  and  structured  set  of  interacting  entities,  which  carry  out  decision 
making  processes  under  constraints,  in  order  to  achieve  a  common  goal. 

The  most  elementary  characteristics  with  respect  to  team  functioning  are  represented  in  Figure 
4.1.  These  characteristics  are  elementary  elements  of  the  functioning  of  a  team.  We  can 
recognise  these  elements  in  the  characterisation  of  a  team  as  noted  by  Dyer  and  Morgan  (Dyer, 
1984,  Morgan,  1986): 

•  a  team  is  a  set  of  two  or  more  individuals; 

•  a  team  has  more  than  one  information  source; 
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•  there  is  interdependence  and  co-ordination  among  members; 

•  there  is  adaptive  management  of  internal  resources; 

•  team  members  share  common  valued  goals; 

•  within  a  team  defined  roles  and  responsibilities  exist; 

•  task  relevant  knowledge  is  available. 


Figure  4.1:  Elementary  Team  Functioning  Characteristics 


All  these  specifications  of  a  team  reoccur  in  the  following  elements  of  team  functioning: 

•  Shared  awareness 

Empirical  data  suggests  (Cannon-Bowers,  1993)  that  teams  employ  a  shared  situational 
awareness.  These  ‘shared  mental  models’  refer  to  organised  knowledge  shared  by  team 
members,  which  can  be  used  to  create  a  context  within  which  decisions  can  be  made  (Orasanu, 
1990).  Another  well-suited  description  for  this  concept  is  Team  mind ’  (Klein  and  Thorsden, 
1989).  As  can  be  seen  in  the  model  (Figure  4.1)  shared  awareness  is  a  cross-section  of  the 
interpretations  of  the  perceived  world  (mental  model)  the  team-members  have.  This  set  of 
‘common’  interpretation  needs  frequent  verification  and  is  controlled  by  the  team  assessment 
procedure. 

•  Team  assessment 

Team  members  are  interdependent  and  work  in  a  co-ordinated  fashion.  This  implies  that  all 
team  members  must  frequently  validate  whether  their  individual  awareness  is  still  compatible 
with  those  of  the  other  team  members.  Failing  to  do  so  would  render  the  decision  making 
process  unreliable,  because  of  differences  in  interpretation  of  their  decision  making 
environment  and  would  consequently  lead  to  errors.  The  team  assessment  also  involves  as  kind 
of  self  appraisal  of  the  team  functioning  and  of  related  individual  functioning.  This  process  of 
self-assessment  is  captured  in  the  team  assessment  process ,  which  may,  if  required,  result  in 
information  redistribution  or  task  reallocation,  in  order  to  establish  a  better  shared  awareness 
and  workload  in  the  team.  This  concept  is  also  referred  to  as  team  maintenance . 
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•  Task  allocation  and  balancing 

Given  the  initial  task  structure,  members  have  been  allocated  specific  tasks.  Due  to  excessive 
workload  or  performance  problems  tasks  can  be  reallocated  to  different  team-members  if 
necessary.  The  dynamic  allocation  of  tasks  is  captured  in  a  task  allocation  and  balancing 
procedure.  This  procedure  (re-)allocates  active  tasks  among  the  team  members  in  such  a  way 
that  workload  is  better  distributed.  This  implies  that  certain  decision  making  processes  should 
be  able  to  be  reallocated.  This  demands  a  certain  level  of  adaptation  in  the  execution  of  tasks  of 
the  individual  team  members. 

•  Information  distribution 

Since  there  is  more  than  one  information  source  within  a  team  and  different  team  members 
may  require  different  information  for  their  specific  tasks  information  distribution  is  an  essential 
characteristic  of  team  functioning.  This  task  can  be  performed  by  human  actors  as  well  as  by 
an  information  management  system  or  by  both.  Inadequate  information  distribution  is  a  cause 
of  many  sub-optimal  functioning  of  teams. 

•  Confirmation  and  authorisation 

Because  of  organisational  structures,  decisions  made  by  team  members  often  have  to  be 
confirmed  and  authorised  by  some  of  the  other  team  members  (or  by  external  decision  makers 
in  which  case  it  falls  outside  the  scope  of  the  team).  The  process  of  confirmation  and 
authorisation  is  usually  an  integral  part  of  military  team  decision  making  and  influences 
decision  making  performance.  It  attributes  to  the  overall  quality  of  decisions  made,  because  of 
constant  verification  and  checking.  It  may  however  disrupt  decision  making  if  the 
organisational  structure  does  not  allow  for  timely  confirmation  or  did  not  take  into  account  to 
create  sufficient  shared  awareness  and  common  information. 

4.2.1  A  DPM  Based  Team  Decision  Process  model 

When  we  combine  the  team  functioning  characteristics  of  Figure  4. 1  with  the  generic  decision 
process  model  of  Figure  4.1  we  get  a  team  decision  process  model.  The  team  decision  process 
model  is  very  much  like  the  decision  process  model,  except  that  it  incorporates  the  team 
functioning  processes  we  introduced  in  the  previous  section.  These  processes,  in  fact,  enable 
the  team  to  function  as  a  decision  making  team  instead  of  a  set  of  independent  decision  makers. 

In  Figure  4.1  we  can  identify  all  the  elementary  team  functioning  characteristics  although  for 
simplicity  we  have  combined  task  allocation  and  task  balancing  in  one  single  process.  We  also 
encounter  an  important  extra  (internal)  information  flow  between  individual  actions  and 
information  distributions. 

Each  team  member  has  his  own  specific  set  of  goals  related  to  his  task  within  the  team  and 
which  either  contributes  to  the  overall  mission  goal  or  to  internal  team  goals.  The  internal  team 
goals  usually  are  meant  to  provide  other  team  members  with  processed  information. 
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Figure  4.1:  Team  Decision  Process  Model 


Furthermore,  the  main  elements  of  the  generic  decision  process  model  are  multiplied  in  the 
team  model.  Each  team  member  is  assumed  to  have  their  own  presented  world,  situational 
awareness,  interpreted  world  and  decision  making  processes  and  consequent  actions.  In  the 
model  these  duplicates  are  denoted  by  multiple  instances  of  that  particular  element. 

4.2.2  Other  Research  on  Team  Decision  Making 

In  section  4.1.4,  we  stated  that  the  decision  process  models  in  the  framework  can  be  specialised 
if  required  or  preferred.  The  same  holds  for  the  team  decision  process  model.  The  model 
provided  in  Figure  4.1  may  be  extended  or  detailed  by  any  model  which  models  team  decision 
making  in  more  detail. 

We  have  studied  the  relations  between  the  TDM  and  other  models  and  found  that  mappings 
can  be  made  between  the  TDM  and  the  other  models.  For  the  length  limitations  of  this  paper 
we  will  not  further  describe  this. 


5.  The  Effect  Level 

In  the  previous  section  we  described  the  main  models  on  decision  processes  as  part  of  the 
conceptual  level.  The  next  step  is  to  describe  the  second  level  in  our  framework  model:  the 
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effect  level.  On  this  level  we  deal  with  how  well  the  decision  process  is  carried  out  and  with  the 
specific  performance  indicators  of  the  decision  process. 

We  have  identified  two  major  concepts  on  this  level:  the  analysis  areas  of  the  decision  process 
and  the  measures  of  merit.  We  will  discuss  the  analysis  area  model  first. 


5.1.1  The  Analysis  Area  Model 


Essential  DP  elements  Essential  TF  elements 


awareness 


shared 

awareness 


Analysis  Areas 


coherence 


gathering 
processing 
distribution 
^  presentation 
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t workload  analysis 
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visualisation 
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Figure  5.1:  Analysis  area  model 


Although  the  two  decision  process  models  of  section  4  can  already  be  used  in  analysing  a 
particular  decision  process  we  aim  to  give  more  guidance  for  the  analysis  process  itself.  In 
order  to  do  that  we  identified  major  areas  of  analysis  within  which  all  the  concepts  of  the 
models  are  included.  Figure  5.1  depicts  these  areas  in  a  schematic  way. 

The  process  of  deriving  these  areas  is  show  in  the  figure.  They  are  based  upon  the  main 
characteristics  within  the  two  models  on  decision  making  and  team  decision  making.  When  we 
combine  similar  or  related  ones  we  end  up  with  the  analysis  areas.  We  can  regard  this  model  as 
an  intermediate  between  the  conceptual  models  and  the  actual  observable  parameters  like  the 
measures  of  performance. 

Each  of  the  analysis  areas  is  related  to  one  or  more  processes  in  the  models  of  decision 
processes.  These  areas  make  up  the  numerous  aspects  that  can  be  distinguished  and  analysed  in 
decision  processes.  Therefore  this  is  an  intermediate  model  which  serves  to  identify  specific 
performance  parameters  of  decision  process  (measures  of  merit)  and  causes  of  sub-optimal 
decision  making  (the  factors  described  in  chapter  6).  The  areas  also  can  be  used  to  derive  more 
specification  tools  to  describe  organisations  and  decision  processes  in  a  somewhat  more 
formalised  way. 
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Information 

This  area  describes  all  characteristics  of  the  input  data  and  information  and  of  data  processing 
functions,  like  total  amount  of  data  and  information,  rate,  frequency,  repetition,  accuracy  and 
uncertainty,  complexity,  availability,  stability  and  or  validity  and  classification  of  information. 
Also  we  find  here  the  distribution  of  data  among  team  members  and  the  actual  presentation  of 
the  data  and  information. 

Internal  knowledge,  skill 

Internal  knowledge  is  that  kind  of  information,  knowledge  or  understanding  which  is  not 
provided  from  the  outside  but  needs  to  be  available  inside  the  process  itself,  either  in  human 
form  or  in  knowledge  repositories.  This  area  can  be  used  to  deal  with  topics  like  skill,  rule  and 
knowledge-based  reasoning  and  with  training  and  education  of  personnel. 

Tasks 

The  task  area  deals  with  the  assigned  tasks  of  team  members,  workload  distribution,  dynamic 
task  reallocation.  This  task  area  has  a  central  place  in  the  analysis  area  model.  When  we  relate 
it  with  information  we  will  have  situations  like  information  overload  for  specific  tasks. 
Combining  it  with  ‘control’  we  can  study  decision  hierarchies,  type  of  organisations,  type  of 
actual  decision  making  like  consensus,  by  veto,  by  majority.  Analysis  methods  are  for  instance 
functional  decomposition  or  cognitive  task  analysis. 

Control 

Control  deals  with  internal  mechanisms  which  monitor  or  check  the  correctness  of  actions,  the 
sequence  of  actions,  the  correctness  of  information,  etc.  They  can  either  be  human-based  (the 
decision  maker  itself)  or  machine-based  (computer  programs).  Interaction  of  human  decision 
makers  with  each  other,  with  system  functions  and  the  hierarchical  organisation  is  part  of  what 
needs  to  be  analysed  here  (e.g.  communication  analysis).  External  control  deals  with  elements 
like  supervisory  control  from  the  outside  of  the  decision  process.  When  in  an  organisation  tasks 
are  being  delegated  to  a  lower  level  a  parallel  control  mechanism  should  be  put  into  place  to 
monitor  and  control  that  the  work  is  carried  out  correctly.  We  have  here  concepts  like 
responsibility,  hierarchical  organisations.  Typical  descriptors  may  be  the  type  of  control  in  a 
team  and  organisation,  consultation,  frequency,  importance,  amount,  means  of  communication, 
etc. 

Decision  selection 

Decision  selection  is  the  mechanism  of  selecting  between  all  possible  and  probable  solutions 
the  actual  (final,  goal  related)  decision.  This  in  itself  is  a  world  of  analysis  we  just  simply 
captured  in  one  single  area.  We  believe  however  that  a  lot  of  processes  related  to  decision 
selection  have  already  been  decomposed  by  our  conceptual  models  and  have  found  a  place  in 
one  of  the  other  analysis  areas.  For  the  moment  we  concentrate  ourselves  therefor  on  issues 
like  selecting  one  decision  among  many  and  the  mechanism  to  do  that,  like  multiple  hypothesis 
calculus,  inference  and  correlation  mechanisms,  pattern  matching,  etc. 


CCRP  Symposium  2002,  June  2002,  Monterey,  CA 


Page  16 


Hans  E.  Keus  A  Framework  for  Analysis  of  Decision  Processes  in  Teams 


Decision  Support 

This  area  deals  with  the  tools  that  are  used  for  the  decision  process.  For  the  analysis  we 
concentrate  on  speed,  correctness,  appropriateness,  etc.  all  resulting  in  an  amount  of  actual 
support  for  specific  decision  process  functions. 

Resulting  from  these  analysis  areas  we  need  to  select  or  design  the  tools  or  procedures  to 
measure  the  performance  of  the  decision  process.  There  are  two  kind  of  tools  we  can 
distinguish:  the  actual  procedural  analysis  methods  to  be  applied  and  the  performance 
indicators  to  be  used  in  these  analysis  methods.  In  this  paper  we  will  describe  only  the  latter  in 
the  next  section. 

5.1.2  Measures  of  Merit 

The  metrics  or  performance  indicators  can  be  used  to  assess  decision  making  in  the  existing 
organisation  and  after  reorganisation  or  addition  of  support  tools.  Within  the  context  of  this 
research,  we  will  denote  these  indicators  with  the  general  term  measures  of  merit  and  they  are 
an  integral  part  of  the  framework.  Like  we  showed  in  the  General  Approach  of  Figure  3.1.  All 
MoM  inside  the  team  we  call  measures  of  performance. 


For  each  of  the  models  in  our  framework,  decision  process,  team  decision  process  and  generic 
support  architecture  separate  measures  of  merit  exist.  Aside  from  these  there  are  also  higher 
order  measures  which  are  applicable.  These  include  indicators  like  reaching  the  mission  goal. 
For  military  organisations  the  term  Measure  of  Force  Effectiveness  (MoFE)  is  used,  following 
the  terminology  of  RTO  TR-9  within  the  context  of  Command  and  Control  system 
assessments.  Within  organisations  however  we  can  distinguish  measures  of  merit  on  different 
levels,  for  example  indicators  that  describe  the  performance  on  individual  level.  This  leads  to 
the  following  hierarchy  of  measures  of  merit  as  shown  in  Figure  5.1. 

The  flow  of  influence  extends  from  individual  performance  and  support  system  performance 
up  to  the  team  performance.  If  performance  on  individual  level  is  low,  then  chances  are  that 
performance  on  higher  level  will  be  affected  as  well. 

Some  constraints  that  any  measure  should  adhere  to  are  the  following: 

•  Is  it  discriminatory ?  That  is,  does  it  identify  real  differences  between  alternatives? 

•  Is  it  measurable ?  That  is,  is  it  possible  to  compute  or  estimate  it? 

•  Is  it  quantitative ?  That  is,  can  it  be  assigned  numbers  or  ranks? 

•  Is  it  objective ?  That  is,  is  it  defined  or  derived  independent  of  subjective  opinion? 

•  Is  it  sensitive ?  That  is,  does  it  reflect  changes  in  system  variables? 
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Figure  5.1:  Levels  of  Measures  of  Merit 

Some  of  these  constraints  are  difficult  to  meet.  We  might  want  to  use  robustness  as  a  measure 
for  example,  but  this  is  a  difficult  concept  to  quantify,  due  the  high  level  of  abstraction  of  the 
term:  it  needs  to  be  broken  up  in  order  to  fulfil  the  characteristics  mentioned  above.  A  concept 
one  might  propose  as  a  measure  of  merit  is  in  fact  an  aggregation  of  other  measures  of  merit  on 
a  lower  abstraction  level.  Robustness  can,  for  instance,  be  broken  up  in  fault  tolerance  and 
redundancy.  In  turn,  these  may  not  satisfy  all  criteria  as  well.  In  short,  there  are  various  levels 
of  measures.  The  key  challenge  is  to  define  measures  at  a  low  enough  level  in  order  to  use 
them  in  analysis. 

Current  work  within  TG006  is  trying  to  derive  the  MoMs  from  the  formalised  models,  relating 
them  to  the  Analysis  Area  Model,  and  to  relate  them  to  specific  analysis  methods.  In  a  later 
paper  we  will  brief  on  these  results. 

6.  The  Causes  Level 

Decision  making  performance  (either  in  a  team  or  by  individuals)  is  heavily  influenced  by  four 
groups  of  factors: 

1.  Operational  context  factors. 

2.  Human  factors 

3.  Organisational  factors. 

4.  Informational  factors. 

Operational  context  factors 

These  factors,  are  imposed  by  the  mission,  the  working  environment  and  situation  encountered. 
They  affect  all  stages  of  the  decision  process  and  are  for  the  most  part  fixed  (i.e.  cannot  be 
easily  altered).  These  factors  would  include  the  presence  of  shifting  or  ill  defined  goals,  time 
constraints  and  uncertain  dynamic  environments.  These  ‘naturalistic’  environmental  factors  are 
described  to  great  extent  in  (Klein,  1993).  Examples  are  rule  of  engagement  (RoE),  doctrine, 
but  also  physical  factors,  like  propagation  conditions,  weather,  etc. 
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Human  Factors 

As  long  as  humans  play  a  substantial  role  in  the  decision  process  the  human  factor  aspects  will 
also  be  a  major  issue  determining  the  quality  of  decision  making.  Aspects  involved  here  are 
work  loads,  team  communication,  skills  and  training  level,  internal  knowledge,  cognitive 
abilities,  adaptive  capabilities,  etc. 

Organisational  factors 

Team  performance  is  heavily  influenced  by  many  organisational  factors.  These  are  factors 
concerning  the  structure  of  the  organisation  and  the  specific  responsibilities  and  decision 
authorisation  definitions.  Some  obvious  factors  are:  the  team  structure,  authorisation 
constraints,  required  procedures,  organisational  hierarchy,  etc. 

Informational  factors 

Information  is  crucial  for  correct  decision  making.  Retrieval  of  adequate  information  can 
become  hampered  in  many  ways.  Information  may,  on  the  other  hand,  be  easily  retrieved  but 
may  be  incorrect  or  incomplete.  These  information-related  problems  may  cause  malfunctioning 
of  the  decision  making  process  itself,  lead  to  errors  in  judgement  and  erroneous  decisions. 
Examples  of  such  factors  are  the  presentation  of  information  and  whether  the  information  is 
complete  and  accurate. 
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Figure  6.1  The  factor  analysis  area  matrix. 


When  we  combine  the  factors  with  the  analysis  areas  described  in  Figure  5.1  we  create  a 
matrix  as  shown  in  Figure  6.1.  This  matrix  can  be  used  for  plotting  measured  problems  and 
analysis  results  in  a  structured  manner.  Not  all  cells  will  be  relevant  in  the  analysis  of  a 
particular  DP,  some  will  even  be  void.  The  matrix  can  be  used  as  a  starting  point  in  order  to 
identify  probable  cause  areas  or  hot  spots.  Subsequent  analysis  can  then  be  executed  in  order  to 
identify  the  actual  problem  causes. 
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7.  The  Solution  Level 

The  intention  of  the  solution  level  is  to  identify  and  specify  solutions  for  improvement  of 
decision  processes.  One  of  the  generic  solutions  we  wish  to  develop  is  a  generic  architecture 
for  decision  support  tools  (DST).  This  is  a  high  level  architecture  which  is  generic  enough  to 
be  applicable  for  a  wide  range  of  decision  support  tools  and  which  can  be  detailed  sufficiently 
to  serve  as  a  the  high  level  design  for  a  specific  DST.  It  should  incorporate  in  its  structure  all 
the  relevant  aspects  to  decision  processes  as  discussed  in  the  previous  sections.  The  results  of 
the  performance  analysis  of  a  decision  process  can  be  used  for  detailing  the  generic  DST 
architecture  in  order  to  arrive  at  a  specific  DST  for  that  decision  process.  Several  methods  of 
deriving  from  the  models  in  the  analysis  framework  a  generic  architecture  are  being 
investigated. 

One  of  the  architectures  under  consideration  is  shown  in  Figure  7.1.  It  is  based  upon 
distinguishing  the  following  categories  of  components: 


•  Information  gathering  components 

These  components  provision  the  support  system  with  appropriate  data  and  information. 
Information  can  be  collected  by  connection  to  databases,  using  agents,  reading  sensory  output, 
etc.  All  the  components  within  this  set  participate  in  the  information  gathering  process 

•  Interaction  management  components 

User  interface  components  provide  the  operator  with  insight  in  the  process,  e.g.  by  offering 
information  to  the  operator  and  receiving  information  the  operator  supplies  to  the  support 
system.  This  can  be  accomplished  by  means  of  display,  voice  or  other  ways  of  communication. 

•  Data  stores  and  information  processing  components 
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Data  stores  hold  all  static  and  dynamic  task  related  information.  Data  stores  include  knowledge 
bases,  profiles,  databases  etc.  Information  processing  components  are  responsible  for 
retrieving,  analysing  and  processing  the  information  stores  in  data  stores.  These  components 
are  used  to  check  data,  deduct  new  information,  perform  complex  calculations  and  filter  out 
irrelevant  information. 

•  Task  management  components 

In  order  to  provide  the  operator  with  appropriate  information,  the  support  system  needs  to  have 
an  idea  of  which  tasks  the  operator  is  performing  at  any  given  time.  This  requires  monitoring 
of  the  actions  of  the  operator  and  insight  in  the  state  of  the  scenario  and  environment.  Using 
this  knowledge  the  task  management  components  can  provide  the  operator  with  support 
information  that  support  his  task  in  progress.  This  kind  of  support  is  therefore  role  and  scenario 
dependent.  These  components  also  control  time  management,  deal  with  progress  monitoring 
and  monitor  and  control  workload  topics. 

Note  that  this  generic  architecture  does  not  specify  how  the  support  system  should  be 
developed,  it  merely  addresses,  in  general  terms,  all  classes  of  components  that  should  be  (and 
usually  are)  included.  The  generic  support  tool  system  is  a  truly  generic  architecture:  every 
support  tool  can  be  projected  on  this  model. 


8.  Decision  Process  Analysis  Flow 

In  the  previous  chapters  we  have  introduced  a  framework  that  includes  models  representing 
several  aspects  of  decision  making.  This  chapter  aims  to  connect  these  models  in  such  a  way 
that  they  can  be  used,  in  a  joint  fashion,  to  analyse  the  decision  making  performance  of  an 
organisation  and  to  generate  suggestions  on  how  to  improve  them.  We  do  not  claim  to  provide 
a  ready-to-use  guideline  for  analysis  in  this  version  of  this  document  yet.  This  section  will 
provide  some  indications  on  how  to  use  the  framework  and  its  models  and  shows  how  to 
integrate  all  elements. 

8.1  Suggested  procedural  approach 

Analysis  of  the  organisation  should  yield  two  results: 

a)  insight  in  the  factors  that  affect  decision  making; 

b)  designs  for  improvement  of  the  decision  making  process. 

In  order  to  achieve  these  goals  the  analysis  framework  can  be  applied  as  illustrated  in  Figure 

8.1. 

The  steps  as  shown  in  Figure  8.1  are  the  following: 

1.  Determination  of  the  major  goals  and  tasks  in  the  particular  problem  domain  or  decision 
process  which  needs  to  be  analysed.  This  first  step  is  elementary  in  guiding  the  analysis 
processes  in  the  framework. 

2.  Determination  of  the  major  characteristics  of  the  DP  and  establishing  boundary  conditions 
for  them.  These  are  the  main  demands  on  the  DP  which  always  have  to  be  met. 

3.  Establish  the  relation  with  the  main  analysis  areas  and  select  the  suitable  analysis  methods. 
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4.  Establish  the  set  of  Measure  of  Merit  and  the  related  analysis  methods  for  these  MoMs. 

5.  Perform  the  actual  analysis. 

6.  Translation  onto  the  factors  model 

7.  Determine  the  solutions  based  on  the  discovered  factors.  Part  of  the  solution  may  be  to 
design  and  develop  specific  decision  support  tools. 


Figure  8.1:  Application  of  the  analysis  framework 

These  steps  enable  us  to  get  insight  in  which  factors  influence  the  decision  processes,  why 
these  factors  affect  the  performance  (moreover,  which  measures  of  merit  they  affect),  and  how 
we  can  devise  support  tools  to  alleviate  the  problems  encountered.  Because  of  the  complex 
nature  of  the  problem  the  sequence  of  steps  may  be  altered  or  steps  may  be  repeated  several 
times  in  order  to  attain  a  usable  representation  of  the  processes  and  resources  involved.  In  fact 
the  steps  may  be  executed  as  a  continuing  cycle,  in  order  to  get  enough  refinement. 


9.  Current  and  Future  R&D 

In  TG006  we  apply  the  Analysis  Framework  described  in  this  document  to  the  case  studies 
mentioned  in  section  2.2.  Already  the  AF  has  gone  through  several  refinement  stages  by  this 
process,  although  we  have  not  finished  that  work  yet.  During  our  research  we  concentrate 
particularly  on  the  following  issues: 

the  completeness  and  suitability  (for  analysis  guidance  purposes  that  is)  of  the  analysis  area 
model; 

the  applicability  of  the  factor-analysis  matrix; 


CCRP  Symposium  2002,  June  2002,  Monterey,  CA 


Page  22 


Hans  E.  Keus  A  Framework  for  Analysis  of  Decision  Processes  in  Teams 


to  correlate  the  MoEs  (especially  team  performance)  with  the  conceptual  models  and  the 
analysis  area  model; 

the  identification  of  specific  dedicated  analysis  methods  and  relate  them  to  the  analysis  area 
model  and  the  MoEs; 

the  derivation  of  a  generic  decision  support  tool  architecture  which  can  be  applied  in  many 
domains  and  should  enhance  interoperability  of  decision  support  systems; 
the  correctness,  coherence,  omissions,  inconsistencies,  ambiguities  in  our  models  and 
concepts  by  using  formal  specification  models; 

The  final  results  of  the  TG006  work  will  be  published  by  the  NATO  RTO  in  a  formal  report  by 
the  end  of  2003. 
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